Client computers can communicate with a server to remotely access information stored at the server. The transfer of information between the server and client computers may be provided by observing standard protocols and using software applications. For example, a hypertext markup language (HTML) browser application at a client computer can communicate over the public Internet using standard communication protocols, such as Transfer Control Protocol/Internet Protocol (TCP/IP) and hypertext transfer protocols (HTTP), to retrieve resources such as web pages or objects from a HTTP server. Web pages may include formatted text as well as objects, such as multimedia elements including, for example, embedded graphics, sounds, and/or video. Exemplary browser applications include Netscape Navigator and Microsoft Internet Explorer.
The America Online (AOL) client is an example of a proprietary browser application. The AOL client executes on a client computer connected to the AOL network. To receive web pages from servers on the AOL network, the AOL client communicates with servers on the AOL network using proprietary communication protocols. When the AOL client seeks to receive web pages available on HTTP servers located on the public Internet, the AOL client communicates with a proxy server on the AOL network. The proxy server then communicates with HTTP servers on the public Internet using standard communication protocols, such as TCP/IP and HTTP, to receive the web pages from the HTTP servers. The web pages received from HTTP servers may be written in a standard language such as HTML, while the AOL client renders web pages written in a proprietary language. Thus, once received, the proxy server translates the web pages, as necessary, into the proprietary language and forwards the translated web pages to the AOL client.
The HTTP specification provides for an HTTP header known as the Referer Header that is intended to reveal the identity of a resource used to call each object included on a displayed web page. More precisely, the HTTP Referer Header specifies “the address [a Uniform Resource Identifier] (URI) of the resource from which the Request-URI was obtained.” See “HyperText Transfer Protocol—HTTP/1.1,” RFC 2616, http://www.w3.org/Protocols/rfc2616/rfc2616.html. An example of the HTTP Referer Header is shown during the processes undertaken when a user navigates to www.aol.com. The rendering of the aol.com web page causes the browser to obtain a number of other objects such as graphics. For each of the items being retrieved the HTTP Referer Header indicates a referring page. For instance, a trace shows the following during the rendering of one of the objects embedded within www.aol.com:
Uniform Resource Locator
(URL):http://m2.doubleclick.net/viewad/698862/roommates180x75.gif
Headers:
Referer: http://ad.doubleclick.net/adi/N2885.AOLcom/B910031.25;sz=180x75;ord=[timestamp]?
Accept-Language: en-us
Interestingly, even though it is the rendering of the aol.com webpage that is initially triggering retrieval of the embedded object, the referrer header identifies a resource other then aol.com. Specifically, the HTTP Referer Header indentifies “ad.doubleclick.net/adi/N2885.AOLcom/B910031.25;sz=180x75;ord=[timestamp]?” as the refering resource.
Generally, the failure of the HTTP Referer Header to identify aol.com results from multiple levels of indirection or redirection needed to ultimately load the embedded objects and the design of the HTTP Referer Header to identify the “lowest” level of redirection rather then the “highest” level of redirection. More particularly, the HTTP specification requires the HTTP Referer Header to indicate the resource from which the requested URL was directly obtained. In keeping with the earlier example, the above trace reveals that the following destination was called prior to calling the embedded object, likely because of an URL embedded within aol.com that specified this destination and resulted on the following request:
URL: http://ad.doubleclick.net/adi/N2885.AOLcom/B910031.25;sz=180x75;ord=[timestamp]?
Headers:
Referer: http://www.aol.com/
Accept-Language: en-us
As can be seen, the resource identified in the HTTP referrer header for the request initially shown above (i.e., http://ad.doubleclick.net . . . ) was retrieved before that first request was sent. And, that resource redirected or otherwise instructed the browser to retrieve the resource at http://m2.doubleclick.net/viewad/698862/roommates180x75.gif. Therefore, the http://ad.doubleclick.net . . . resource is the resource from which the embedded http://m2.doubleclicknet . . . resource was obtained, causing the HTTP Referer Header to identify http://ad.doubleclick.net . . . rather than www.aol.com, which is the URL for the web page whose rendering triggered the embedded resource to be retrieved.
Notably, implementations of the HTTP referrer handles frame pages by including the URL of the web page loaded into a frame, rather than the URL of the frame page.